A Hybrid Model for Spontaneous Speech Understanding

This paper presents a hybrid model to understand spontaneous speech by combination of speech connotation and denotation analyses. The testbed of the approach is an intelligent tutoring system (ITS) collected by Wizard-of-Oz (WoZ) simulations. The children users are inexperienced and their utterances are often dysfluent and have loose grammar structure. To robustly understand spontaneous speech in the tutorial environment, we categorize the user utterances into 30 tutoring events, which can reflect the content meaning of utterances in a broad and shallow way. The objective of this study is to classify utterances into the target events. The tutoring event classification integrates speech connotation analysis and speech denotation analysis. The speech connotation analysis intends to model the cognitive states of students by three classes: confidence, puzzlement, and hesitation. The speech denotation analysis intends to compute the event-utterance similarity based on the TF•IDF vector of some pragmatically and semantically salient words embedded in the utterances. We define salient words by those words that contain novel information neither presupposed by the interlocutor nor denoted in the precedent part of the utterance. We used speech and transcribed text for experiments, and achieved 75.5% accuracy when the salient words were manually annotated. The accuracy reduced by 15.4% relative when the salient words were automatically extracted.